Microsoft confirms Reddit blocked Bing Search

Reddit updated its robots.txt file prohibiting Bing and some other search engines from crawling its site.

Chat with SearchBot

Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site. “Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” a Microsoft representative told Search Engine Land.

What happened. On July 1, 2024, Reddit updated its robots.txt file preventing many search engines and AI tools from crawling the site. Reddit did not prevent Google from crawling the site, despite what some people may have thought earlier this month. But Reddit did prevent most other crawlers from crawling the site.

Earlier this morning, Mark Williams-Cook notified me that Reddit results were dropping out of the Bing Search index. Then several media outlets began covering the news. I wanted to confirm that Bing’s crawlers were indeed blocked, because Reddit was using IP detection to show search engines one version of its robots.txt file and humans another version of the robots.txt file – as I explained earlier this month.

Bing has thus stopped crawling new content on Reddit, which is why when you filter Reddit results in Bing Search for the last week, you see nothing:

Bing Reddit Index Last Week

Microsoft confirmed. A Microsoft spokesperson told Search Engine Land:

“Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”

Reddit statement. Reddit spokesperson Tim Rathschmidt says in a statement to The Verge:

“This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”

Why we care. With Reddit securing a licensing deal with Google, Reddit is able to play hardball with other search engines and AI tools. So Reddit has blocked most other search engines from crawling its content. Meanwhile, Google is driving insane traffic to Reddit these days, including testing special treatment in its search results for them.

It makes you wonder if other large websites can try to go down this route and where that might leave smaller publishers and content producers.

Meanwhile, don’t expect to see much new Reddit content being surfaced on Bing in the near future.


About the author

Barry Schwartz
Staff
Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics.

In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the "US Search Personality Of The Year," you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O'Clock.

Barry can be followed on X here and you can learn more about Barry Schwartz over here or on his personal site.

Get the newsletter search marketers rely on.